Lectures on Nonparametric Bayesian Statistics

Notes for the course by Bas Kleijn, Aad van der Vaart, Harry van Zanten (Text partly extracted from a forthcoming book by S. Ghosal and A. van der Vaart) version 4-12-2012 UNDER CONSTRUCTION 1 Introduction Why adopt the nonparametric Bayesian approach for inference? The answer lies in the simultaneous preference for nonparametric modeling and desire to follow a Bayesian procedure. Nonparametric (and semiparametric) models can avoid the arbitrary and possibly un-verifiable assumptions inherent in parametric models. Bayesian procedures may be desired for the conceptual simplicity of the Bayesian paradigm, easy interpretability of Bayesian quantities or philosopohical reasons. 1.1 Motivation Bayesian nonparametrics is the study of Bayesian inference methods for nonparametric and semiparametric models. In the Bayesian nonparametric paradigm a prior distribution is assigned to all unknown quantities (parameters) involved in the modeling, whether finite or infinite dimensional. Inference is made from the " posterior distribution " , the conditional distribution of all parameters given the data. A model completely specifies the conditional distribution of all observable given all unobserved quantities, or parameters, while a prior distribution specifies the distribution of all unobservables. From this point of view, random effects and latent variables also qualify as parameters, and distributions of these quantities, often considered as part of the model itself from the classical point of view, are considered part of the prior. The posterior distribution involves an inversion of the order of conditioning. Existence of a regular version of the posterior is guaranteed under mild conditions on the relevant spaces (see Section 2). 1.1.1 Classical versus Bayesian nonparametrics Nonparametric and semiparametric statistical models are increasingly replacing parametric models, to overcome the latter's inflexibility to address a wide variety of data. A nonpara-metric or semiparametric model involves at least one infinite-dimensional parameter and hence may also be referred to as an " infinite-dimensional model ". Indeed, the nomenclature " nonparametric " is misleading in that it gives the impression that there is no parameter in the model, while in reality there are infinitely many unknown quantities. However, the term nonparametric is so popular that it makes little sense not to use it. The infinite-dimensional parameter is usually a function or measure. In a canonical example of nonparametric model the data are a random sample from a completely unknown distribution P. More generally, functions of interest include the cumulative distribution function, density function, regression function, hazard rate, transition density of a Markov process, …